chore: support pattern as pipeline key name
#4368
Conversation
WalkthroughThe updates make significant refinements to the ETL pipeline in the project. Primarily, the changes involve enhancing pattern processing capabilities for dissecting and regex functionalities, introducing new methods and constants, and improving test coverage to ensure the correctness of these features. Additionally, test files were refactored to use a common function for column schema creation, promoting code reusability and simplifying the codebase. Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
shuiyisong
changed the title
pattern as pipeline key name with patternspattern as pipeline key name
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (8)
- src/pipeline/src/etl/processor/dissect.rs (2 hunks)
- src/pipeline/src/etl/processor/regex.rs (3 hunks)
- src/pipeline/tests/common.rs (2 hunks)
- src/pipeline/tests/dissect.rs (1 hunks)
- src/pipeline/tests/gsub.rs (2 hunks)
- src/pipeline/tests/join.rs (1 hunks)
- src/pipeline/tests/on_failure.rs (5 hunks)
- src/pipeline/tests/regex.rs (1 hunks)
Additional comments not posted (13)
src/pipeline/tests/gsub.rs (1)
16-16: LGTM!The changes improve code reuse by using the
make_column_schemafunction from thecommonmodule.src/pipeline/tests/common.rs (1)
15-15: LGTM!The addition of the
SemanticTypeparameter to themake_column_schemafunction enhances the accuracy of the test schema.src/pipeline/tests/join.rs (1)
35-44: LGTM!The changes improve code reuse by using the
make_column_schemafunction from thecommonmodule.src/pipeline/tests/regex.rs (1)
1-109: LGTM!The changes improve code reuse by using the
make_column_schemafunction from thecommonmodule.src/pipeline/tests/dissect.rs (2)
20-65: LGTM!The test function
test_dissect_patternis well-structured and validates the functionality as expected.
67-113: LGTM!The test function
test_dissect_patternsis well-structured and validates the functionality as expected.src/pipeline/tests/on_failure.rs (4)
Line range hint
16-48: LGTM!The test function
test_on_failure_with_ignoreis well-structured and validates the functionality as expected.
Line range hint
49-93: LGTM!The test function
test_on_failure_with_defaultis well-structured and validates the functionality as expected.
Line range hint
94-129: LGTM!The test function
test_defaultis well-structured and validates the functionality as expected.
Line range hint
130-181: LGTM!The test function
test_multiple_on_failureis well-structured and validates the functionality as expected.src/pipeline/src/etl/processor/regex.rs (2)
160-162: LGTM!The method
try_with_patternsis well-structured and handles errors appropriately.
216-243: LGTM!The test function
test_simple_parseis well-structured and validates the functionality as expected.src/pipeline/src/etl/processor/dissect.rs (1)
562-565: LGTM!The method
try_fromforDissectProcessoris well-structured and handles errors appropriately.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4368 +/- ##
==========================================
- Coverage 85.21% 84.87% -0.34%
==========================================
Files 1060 1063 +3
Lines 189484 190148 +664
==========================================
- Hits 161474 161395 -79
- Misses 28010 28753 +743 |
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
This pr mainly
patternas field key name in pipeline yaml along withpatternsChecklist
Summary by CodeRabbit
New Features
Tests